Incubate persistent data structures #31

msteindorfer · 2021-04-30T10:01:50Z

This work-in-progress draft PR kicks the tires for incubating persistent data structures into the swift-collections project.

A long-term goal of this effort is to mature persistent replacements for the general-purpose data structures already present in the Swift Standard Library (i.e., Array, Set and Dictionary).

Currently, the PR covers the HashMap<Key: Hashable, Value> data type, which is supposed to become a persistent drop-in replacement for Swift's Dictionary. The draft implementation is neither feature complete nor performance optimized yet, but aims to create visibility for this effort and to elicit early feedback. Working title for the module is Capsule, based on prior work.

HashMap<Key: Hashable, Value> implements the Compressed Hash-Array Mapped Prefix-tree (CHAMP) data structure. The implementation is inspired by previous JVM data structures that I've authored in the past, amongst them:

Scala's HashMap and HashSet (Scala; default in standard library since Scala 2.13)
The Capsule Hash Trie Collections Library (Java; standalone library)

Feel free to provide feedback on idiomatic Swift usage and Swift-specific performance optimization tips and tricks, if and where appropriate.

References and Further Readings

Publications

Talks

Checklist

I've read the Contribution Guidelines
My contributions are licensed under the Swift license.
I've followed the coding style of the rest of the project.
I've added tests covering all new code paths my change adds to the project (if appropriate).
I've added benchmarks covering new functionality (if appropriate).
I've verified that my change does not break any existing tests or introduce unexplained benchmark regressions.
I've updated the documentation if necessary.

lorentey · 2021-05-04T20:23:14Z

This is exciting work. I think these would be a great fit for this package, and I'm personally very much interested in getting these landed (& eventually added to the stdlib). We'll need to figure out how to test & benchmark these, and carefully evaluate API decisions. I think there is a bunch of interesting low-level optimization work we can do on the internal representations (things like replacing Array<Any> storage with tail-allocated typed buffers).

(I think it may make sense to open an official feature branch for this project. I'll ponder how best to coordinate work, and I'll see about scheduling resources for getting this done.)

msteindorfer · 2021-05-06T16:02:07Z

Hi @lorentey, great that we share the same interest and vision regarding this effort.

(I think it may make sense to open an official feature branch for this project. I'll ponder how best to coordinate work, and I'll see about scheduling resources for getting this done.)

An official feature branch sounds good to me. Keep me up to date on how we should progress incubating the persistent collections.

(The draft PR was a suggestion by @kylemacomber and was actually a good thing for getting the draft implementation out and creating momentum. Though it may not be ideal acting as a long running feature branch.)

We'll need to figure out how to test & benchmark these, and carefully evaluate API decisions.

Sure. I plan to add more test coverage (and eventually benchmarks) for the persistent data structures along the way.

I didn't investigate the state of the conformance checkers in swift-collections yet, but having good coverage for Dictionary, Set, Array and entailed protocols would help a lot.

I'll also reach out to discuss API decisions and idiomatic Swift usage when time is due, but first I'll work on dialing in the data structure design (e.g., what concepts translate best to Swift) and feature completeness.

I think there is a bunch of interesting low-level optimization work we can do on the internal representations

The same goes for low-level optimizations. I'm looking forward to tackle those as well, but first the data structure design needs to settle. Feel free to make suggestions anytime if you happen to notice something.

things like replacing Array storage with tail-allocated typed buffers

The Array<Any> store disguises a lot of complexity, since its content is type-heterogeneous, akin to several (differently) typed buffers concatenated (therefore elements are currently typed as Any).

Can you provide me some references on what you had in mind? I will then have a look at it.

ktoso · 2021-05-11T07:58:05Z

Oh man, this is great~! Fantastic to have your expertise here @msteindorfer! 🥳
I regularly miss persistent data structures so this'll be a lovely and very welcome addition!

Sources/Capsule/_Common.swift

WowbaggersLiquidLunch · 2021-10-12T10:36:39Z

Sources/Capsule/HashMap.swift

+            guard expectedCount == builder.count else {
+                preconditionFailure("Duplicate key: '\(key)'")
+            }


I think this can be replaced with

precondition(expectedCount == builder.count, "Duplicate key: '\(key)'")

Although, I don't know if there is any performance disadvantage with the auto closure in precondition.

WowbaggersLiquidLunch · 2021-10-12T10:43:43Z

Sources/Capsule/HashMap.swift

+    public func get(_ key: Key) -> Value? {
+        rootNode.get(key, computeHash(key), 0)
+    }


I think it's more idiomatic in Swift to name this function as value(at key: Key) -> Value?.

Also, maybe this function can be internal or private? There is already a subscript for retrieving the value by key.

kielgillard · 2022-06-14T06:27:00Z

@lorentey @msteindorfer is there anything we in The Community can do to help get this along?

msteindorfer · 2022-06-15T12:48:39Z

@kielgillard, this contribution will be actively worked on to be readied for inclusion. Stay tuned.

Correctly reflects refining edits as well and not only initial draft date.

…ble` skeleton

…skeleton

Further add missing items to the benchmark library.

…ible`

Note, always tracking discardable result negatively impacts batch use cases.

Necessary due to removal of unchecked initializer variants.

lorentey · 2022-09-07T16:18:17Z

I'm landing this on a feature branch to start integration & performance work. 💯

msteindorfer force-pushed the capsule-collections branch from 997a1f6 to 9ab9563 Compare May 4, 2021 07:14

msteindorfer force-pushed the capsule-collections branch from 1866b04 to 62658f8 Compare May 15, 2021 10:21

msteindorfer force-pushed the capsule-collections branch 2 times, most recently from af069c9 to 37e957f Compare June 4, 2021 15:03

xwu reviewed Jun 19, 2021

View reviewed changes

Sources/Capsule/_Common.swift Outdated Show resolved Hide resolved

WowbaggersLiquidLunch reviewed Oct 12, 2021

View reviewed changes

msteindorfer force-pushed the capsule-collections branch from 9919993 to 1955a63 Compare November 29, 2021 11:08

lorentey mentioned this pull request Dec 9, 2021

Counted Set #132

Draft

7 tasks

msteindorfer force-pushed the capsule-collections branch from 1955a63 to bd6a864 Compare June 15, 2022 19:00

msteindorfer added 15 commits August 9, 2022 17:20

[Capsule] Import sources

488322e

[Capsule] Add package configuration

449e570

[Capsule] Adhere to CollectionsTestSupport in test code

13bd00b

[Capsule] Increase visibility of count and subscript to public

03913fd

[Capsule] Update copyright headers

097e6b7

[Capsule] Split HashMap extensions into separate files

3cb6fb3

[Capsule] Rename Common.swift -> _Common.swift

8805172

[Capsule] Split nested structs and classes into separate files

1598e22

[Capsule] Harmonize identation

b3e1e9d

[Capsule] Update copyright headers

ce5a6bd

Correctly reflects refining edits as well and not only initial draft date.

[Capsule] Remove unused XCTest import

29dd2c8

[Capsule] Implement Hashable extension for HashMap

b518d56

[Capsule] Rename and simplify smoke tests

77ef612

[Capsule] Implement preliminary hash-collision resolution

e09c8db

[Capsule] Make spine of trie monomorphic

d4ecaa0

msteindorfer added 23 commits September 2, 2022 12:19

[Capsule][WIP] Return to monomorphic spine

26893b1

[Capsule] Add `BitmapIndexedDictionaryNode: CustomDebugStringConverti…

0550c27

…ble` skeleton

[Capsule] Add BitmapIndexedDictionaryNode: CustomStringConvertible …

81a38b3

…skeleton

[Capsule] Add smoke test for larger hash-collision nodes

ef523a9

[Capsule][WIP] Return to monomorphic spine, continued

bf22461

[Capsule] Enable Keys and Values view benchmarks

8bc5e20

[Capsule] Use non-materializing data/trie slice views in iterator

45b9d8d

[Capsule] Rename base iterators

e9dec67

[Capsule] Remove not necessary type alias

870e49d

[Capsule] Add init(uncheckedUniqueKeysWithValues:) benchmark

f6fe3c0

Further add missing items to the benchmark library.

[Capsule] Fix TODO comment

a4bce51

[Capsule] Implement `BitmapIndexedDictionaryNode: CustomStringConvert…

6b905eb

…ible`

[Capsule] Rename package to PersistentCollections

5bc17ff

[Capsule] Simplify insertion methods

4e213be

[Capsule] Simplify deletion methods

7ea47d7

[Capsule] Unify insertion/deletion methods

26b97f9

Note, always tracking discardable result negatively impacts batch use cases.

[Capsule] Remove unchecked variant of initializer

49283d3

[Capsule] Remove comment after check

868eb2f

[Capsule] Fix test case argument label

031b762

Necessary due to removal of unchecked initializer variants.

[Capsule] Add subscript(position:) benchmarks

e2f5344

[Capsule] Expand index(forKey:) tests

f930801

[Capsule] Remove non-applicable tests

fbb8aed

[Capsule] Clean-up annotations

ccd11e5

lorentey marked this pull request as ready for review September 7, 2022 16:15

lorentey self-requested a review as a code owner September 7, 2022 16:15

lorentey changed the base branch from main to feature/PersistentCollections September 7, 2022 16:16

lorentey merged commit de5fad7 into apple:feature/PersistentCollections Sep 7, 2022

lorentey added the HashTreeCollections Hash-array mapped prefix tree implementations label Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incubate persistent data structures #31

Incubate persistent data structures #31

msteindorfer commented Apr 30, 2021

lorentey commented May 4, 2021

msteindorfer commented May 6, 2021

ktoso commented May 11, 2021

WowbaggersLiquidLunch Oct 12, 2021

WowbaggersLiquidLunch Oct 12, 2021

kielgillard commented Jun 14, 2022

msteindorfer commented Jun 15, 2022

lorentey commented Sep 7, 2022

Incubate persistent data structures #31

Incubate persistent data structures #31

Conversation

msteindorfer commented Apr 30, 2021

References and Further Readings

Publications

Talks

Checklist

lorentey commented May 4, 2021

msteindorfer commented May 6, 2021

ktoso commented May 11, 2021

WowbaggersLiquidLunch Oct 12, 2021

Choose a reason for hiding this comment

WowbaggersLiquidLunch Oct 12, 2021

Choose a reason for hiding this comment

kielgillard commented Jun 14, 2022

msteindorfer commented Jun 15, 2022

lorentey commented Sep 7, 2022